[BEAM-4783] Fix issues created in #6181.#7690
Closed
kyle-winkelman wants to merge 1 commit intoapache:masterfrom
Closed
[BEAM-4783] Fix issues created in #6181.#7690kyle-winkelman wants to merge 1 commit intoapache:masterfrom
kyle-winkelman wants to merge 1 commit intoapache:masterfrom
Conversation
Member
|
Thanks @kyle-winkelman sorry for not responding before to your messages. I was aware of the issue, just too busy with other stuff to take a look again. I will do and ping you when done. |
iemejia
approved these changes
Feb 8, 2019
Member
iemejia
left a comment
iemejia
left a comment
There was a problem hiding this comment.
LGTM sorry for the delay, and thanks for bringing back this fix @kyle-winkelman.
I am not sure there is going to be a new RC but if there is I will try to include this one. Anyway next version (2.11.0) is going to be cut next week so it will be included there for sure.
iemejia
added a commit
that referenced
this pull request
Feb 8, 2019
Member
|
Merged manually to correct the commit title and add an extra |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix issues created in #6181 and incorrectly fixed in #6884 (although this PR increased readability greatly).
Before any work on BEAM-4783, GroupCombineFunctions would always use a
new HashPartitioner(rdd.rdd().sparkContext().defaultParallelism());.The intent was to skip the creation of this
Partitionerand callgroupByKey()with no arguments only when the new bundleSize option was in use. #6181 actually did the opposite causing performance impacts, and because #6181 had terrible readability #6884 did not fix it correctly.I am also hopeful that this can be cherry picked into 2.10 if another RC is created to get the Spark Runner's performance back to the levels seen in 2.7.
@iemejia @timrobertson100
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.It will help us expedite review of your Pull Request if you tag someone (e.g.
@username) to look at it.Post-Commit Tests Status (on master branch)